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Abstract 


This report analyzes evidence from a new study assessing the effectiveness of Quantway 1 
(QW1), a single-term accelerated developmental math course with a focus on quantitative 
reasoning. Quantway® is one of the Pathways courses (along with Statway”) developed by a 
network of faculty and content experts convened by the Carnegie Foundation to accelerate the 
progress of college students through developmental mathematics with the aim of increasing 
success rates in earning college credit in mathematics. 

This report is the sequel to an earlier study’ of the effectiveness of QW1, both of which 
applied a multilevel propensity score matching approach. The first study provided robust 
evidence that QW1 increases student success in fulfilling developmental math requirements 
and advances equity in student outcomes. The present study tracked student college math 
achievement through the year after QW1 enrollment. Results in this most recent analysis 
revealed that QW1 students were significantly more likely to enroll in credit-bearing college 
math courses within a year than their counterparts who followed traditional developmental 
math sequences were, and that while increasing subsequent math course taking, QW1 students 
demonstrated a comparable GPA. Significantly, QW1 effects were positive across all sex and 
race/ethnicity subgroups, as well as in nearly all classrooms and colleges. The current study 
provided additional empirical evidence of the persistence of QW1’s effectiveness for diverse 
student populations across varied classroom and institutional contexts. Directions for future 
work are discussed. 

As indicated in the prior study, additional questions still need to be answered: Do the 
effects of QW1 persist in the following year? More specifically, we are interested in whether or 
not QW1 students are (a) more likely than matched comparison students to enroll in credit- 
bearing college math courses, and (b) perform comparably to or better than matched 
comparison students in college math courses. Because QW1 is designed not only to get 
students through their developmental math sequences but to help them meet their college- 
level math requirements, these questions will be particularly important to answer in 
determining QW1’s effectiveness. Therefore, the objective of the current study is to assess the 
persistence of effects of QW1 by tracking student college math achievement one year after 
QW1 participation. 


Method 

Propensity Score Matching 

Propensity score matching is a statistical technique applied to observational data to 
reduce possible selection bias—where certain kinds of students may have been more likely to 
enroll in QW1, leading to more positive outcomes than there otherwise would have been—and, 
accordingly, increase the validity of causal inference (Rosenbaum & Rubin, 1983). There are two 
main steps involved in this procedure: (a) obtain a propensity score per student, which 
represents the likelihood of a student enrolling in QW1, and (b) identify as matches students 
whose propensity scores are similar to each other. Typically, a logistic regression approach is 


* See Yamada, H., Bohannon, A., & Grunow, A. (2016) 
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used with a set of factors or covariates hypothesized to influence student enrollment in QW1. 
To obtain propensity scores in the earlier study (Yamada et al., 2016), we selected a total of 37 
student-level covariates including student background characteristics and prior course taking 
and success patterns during the two years prior to the Quantway 1 term based on prior 
research findings and advice from institutional researchers in the participating colleges and ran 
a hierarchical linear modeling (HLM) approach (Hong & Raudenbush, 2005, 2006; Raudenbush 
& Bryk, 2002; Yamada & Bryk, 2016). 

In step two, we conducted propensity score matching separately for each cohort and 
college by applying a nearest neighbor matching algorithm (Rosenbaum & Rubin, 1985). This 
algorithm was appropriate for our study because we wanted to retain as many Quantway 1 
students as possible and had a large pool of non-Quantway 1 students for creating matches. We 
attempted to find up to five matches per Quantway 1 student (5:1 ratio matching) to maximize 
the best matches from the non-Quantway 1 student group while still maintaining precision 
(Ming & Rosenbaum, 2000). We also specified a caliper distance of up to 0.2 to reduce the risk 
of bad nearest neighbor matches based on recommendations in the literature (Austin, 2011; 
Rosenbaum & Rubin, 1985). For propensity score matching, we used the package Matchlt (Ho, 
Imai, King, & Stuart, 2011) in R (R Core Team, 2015). For more details regarding propensity 
score matching see Yamada et al. (2016). 


Analytic Sample 

In our earlier matching study, we identified a total of 16,440 students (3,992 QW1 
students and 12,448 matched comparison students) across 10 colleges, leading to an average of 
approximately three matched students for each QW participant (Yamada et al., 2016). In the 
current study, however, we had to reduce the analytic sample to 10,184 students (2,406 QW1 
students and 7,778 matched comparison students) across nine colleges due to the 
unavailability of course-taking data after QW1 enrollment. The ratio of program students to 
comparison group students remained the same at 1:3. 


Study Design 

As student follow-up measures, we looked at (a) student enrollment in college-level 
math courses in the subsequent calendar year, including a summer term where applicable (e.g., 
tracking student math course enrollments over the spring, summer, and fall terms for the fall 
cohorts), and (b) their corresponding GPA’ in any math courses taken. We tracked these two 
measures right after QW1 enrollment for QW1 students. For matched comparison students, we 
tracked the same outcomes over the same time period right after (a) they had successfully 
completed their developmental math sequences (e.g., completing the requirements in one 
semester), or (b) the entire academic year regardless of their success in developmental math 
sequences. 

Similar to the previous study (Yamada et al., 2016), we applied a four-level hierarchical 
linear modeling (HLM) approach (Raudenbush & Bryk, 2002), in which matched clusters (level 1) 
were nested within QW1 students (level 2), who were in turn nested within QW11 faculty 


* We assigned a value of 0 as a a grade point to W (Withdrawal) and | (Incomplete) in this analysis in order to create a conservative metric while 
allowing us to maintain all matched students. 
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member classrooms (level 3) within their colleges (level 4). We estimated QW1’s effectiveness 
by comparing student enrollment rates of QW1 students with their matched comparisons with 
a binary outcome. Enrollment was defined as at least one attempt at a college math course. For 
the analysis of college math GPA, we treated it as a continuous outcome and estimated an 
effect of QW1 on the GPA. We used HLM 7 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 
2011) for all of the HLM analyses. 


Analytic Approach 

College Math Enrollment 

To estimate differences in college math enrollment rates, we constructed a four-level 
Bernoulli model and estimated its model parameters using maximum likelihood via penalized 
quasi-likelihood estimation. jx represents the probability that student / matched with QW1 
student j associated with faculty member k’s class in college / successfully completed the 
developmental math sequence. Correspondingly, nix is the corresponding log-odds of this 
outcome and formally expressed as: 


Level 1 Model (Matched cluster) 
Prob(ATT jx=1 | Mxi) = Dik, 
log[Dixi/(1 - Dixi)] = Niky 
Nigkt = Tlojki + Tayx® (QW iki) + + Tajkr*(PSijt), 


Level 2 Model (QW1 Student) 
Tlojki = Books + Borki(W12jx)) + BorK(S12 jx) + Boski(F12 jx) + Boaki(S13 x1) + Fojki 
Teyki = Bros + Br1ki(W12 ji) + Br2K(S12 jc) + B13K(F12 jc) + Braki(S13 x1) + Fajki 
Tejk1 = B20K1, 


Level 3 Model (Faculty) 
Book! = ooo! + Uooki, 


Borki = Voor 
Bo2ki = Vo2oh 
Bo3ki = Vo30) 
Boaki = Voao, 


B10Ki = Vi001 + Ur0ki, 
611K = V1101 


B12ki = V120h 
613ki = V130/ 
B1aki = V1401, 
6B 20ki = Y2001, 


Level 4 Model (College) 
Vooo! = Ooov0 + Vooo! 
Yo10/ = 80100, 

Yo20! = 0200, 
Yo30/ = 60300, 


CARNEGIE POST-QUANTWAY SUCCESS 5 


Voaoi = Oo400, 
V1001 = ©1000 + Vi00) 
V1101 = 61100, 
¥1201 = 61200, 
¥1301 = 61300, 
¥1401 = 61400, 
¥2001 = 62000, 


where ATT represents college math enrollment (1 for having attempted at least one course and 
0 for not having attempted), and QW is a dummy variable indicating whether the student was 
enrolled in QW1 (coded as 1) or one of the matched comparisons (coded as 0). As a further 
safeguard, we included individual students’ propensity scores, PS, as an additional adjustment 
variable. W12 to $13, which are dummy variables for the four cohort groups with Fall 2013 as a 
reference category, were included at Level 2 as additional adjustment variables for the 
outcome. 


Results 


The analysis indicates that QW1 students are more significantly more likely to take an 
additional college math course compared to their peers in remedial math classes. The findings, 
illustrated in Table 1, indicate, on average, QW1 students demonstrated significantly higher 
odds of enrollment, 2.33 (95% Cl [1.49, 3.66]°), suggesting that the odds of QW1 students 
taking at least one college math course was 2.33 times as large as the matched comparison 
students.’ The corresponding estimated probabilities of enrollment were 50.35% for the QW1 
group and 30.31% for the matched comparison group, suggesting that about half of the QW1 
students enrolled in at least one college math course in the subsequent year, and that less than 
a third of the matched comparison students did so. The estimated correlations between the 
intercept and the slope at both college and faculty levels were negative (-.55 and -.81, 
respectively), suggesting that the lower the college math enrollment rate for the matched 
comparison group, the larger the effect of QW1, and that this tendency was stronger among 
classrooms than among colleges. Effectively, QW1 students at schools where traditional 
remedial math students are less likely to take college level math were, in fact, even more likely 
than their peers to take additional math classes. In addition, we found variation (0.299 and 
0.094 for the college and faculty variances) in QW1 effect among colleges and faculty members. 
Figures 1 and 2 display the variation in QW1 effect size at the college and faculty levels, 
respectively. In both charts, we added three lines as references. The center line represents the 
average effect of QW1, and the upper and lower lines represent the upper and lower bounds of 
the average effect (which are deviated in two SEs from the center line). A value of 0 in logits 
means no QW1 effects. Figure 1 demonstrates that there were positive QW1 effects on student 


> HLM 7 generates 95% confidence intervals of odds ratios. 

“We also conducted sensitivity analyses (Hong & Raudenbush, 2005, 2006) on a QW1 effect on college math enrollment. Results indicated that 
with an adjustment for the largest potential hidden bias, the 95% confidence interval for the new QW1 effect estimates did not contain 0 or any 
negative values, thereby supporting the strong ignorability assumption. Thus, it is very unlikely that our general conclusion regarding the 
positive effect of QW1 on the student outcome has been influenced by the omission of unmeasured confounding factors. 
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outcomes in all colleges. College 8 stands out as a positive deviant with a QW1 effect outside 
the upper bound of the average effect. Figure 2 shows the variation in QW1 effectiveness 
across the classrooms in the NIC. The vast majority of QW1 faculty at College 8 drastically 
outperformed the average QW1 faculty, suggesting internal coherence at this institution. In 
contrast, a wide range of variation was observed among faculty members at College 3. These 
patterns in Colleges 8 and 3 were consistent with those found in the previous study in the size 
of the QW1 effect on fulfilling developmental math course requirements (Yamada et al., 2016). 


College Math Performance 

To estimate differences in college math GPAs, we constructed a four-level normal model 
and estimated its model parameters using maximum likelihood via full maximum likelihood 
estimation. CMGPAju represents college math GPA that student i matched with QW1 student / 
associated with faculty member k’s class in college / earned and is formally expressed as: 


Level-1 Model (Matched cluster) 
CMGPA ji = Tojki + Trjta™* (PSijti) + Tojx1* (QWEC iia) + T3yx*(CMATTE Cixi) 
+ Tajr* (CMINT jxi) + Cijxl, 


Level-2 Model (QW1 student) 
Teojkt = Book + Borki*(W12jx)) + Borki*(S12jki) + Boski* (F12jx1) + Boan* (S13 jx) + Fojkl, 
Terjki = Br0K1, 
Tjki = Bros + B21K/*(W12jx)) + B224* (S12 jx) + B23" (F12 jx) + B2ax*(S13jxi) , 
Tei = Baoxi + B31K/*(WI12jx)) + B324* (S12 jx) + B33xi*(F12 jx) + B3au*(S13jxi) , 
Taji = Bao + Barki*(W12jx)) + Bazi* (S12 jk) + Basc*(F12 jx) + Baau* (S13 jxi) , 


Level-3 Model (Faculty) 
Boo! = Yoo! + Uooki, 
Bo1ki = Vo1oh 
Boaki = Yo20r 
Bo3ki = Vo30h 
Boaki = Voaor 
B 10k! = Vi00/ 

B20k1 = V200h 
B21ki = V210h 
B22Ki = Y220h 
6B 23ki = 230) 
B2aki = ¥2401 
B30ki = V300/ 
B31ki = 3101 
632i = 320) 
6B33k/ = 330) 


° We also ran a series of random slope models. However, we observed high correlations involved in the slopes and the intercept, which 
suggested a fixed slope model. Results from those random slope models were very similar to those from the fixed slope model. 
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B3aki = V340) 
Baoki = Vaooh 
Barki = Varo) 
Bazki = Va201 
Ba3ki = Va301 
Baaki = Vaaor, 


Level-4 Model (College) 
Yooo! = Ooov0 + Voooi, 
Yo10/ = 80100, 
Yo20! = 0200, 
Yo30/ = 90300, 
Yoao! = Oo400, 
V100/ = 1000, 
¥200/ = 62000, 
V210/ = 62100, 
¥2201 = 62200, 
¥230/ = 62300, 
V240/ = 62400, 
V300/ = 63000, 
¥310/ = 63100, 
¥3201 = 63200, 
¥330/ = 63300, 
V3401 = 63400, 
Vaoo! = O4000, 
Va1ol = 64100, 
¥a201 = 64200, 
¥4301 = 64300, 
Vaaol = 64400, 


where QWEC indicates whether the student was enrolled in QW1 (coded as 1) or one of the 
matched comparisons (coded as -1), CWATTEC represents college math enrollment (1 for 
having attempted at least one course and -1 for not having attempted), and CMINT is an 
interaction term of these two variables. We applied effect coding to the grouping variables in 
order to directly represent both main and interaction effects on the outcome. As a further 
safeguard, we included individual students’ propensity scores, PS, as an additional adjustment 
variable. W12 to $13, which are dummy variables for the four cohort groups with Fall 2013 as a 
reference category, were included at Level 2 as additional adjustment variables for the 
outcome. 

Overall, QW1 students had similar GPAs to other remedial math students who 
completed college level math courses. Our main focus was on the interaction effect between 
QW 1 and college math enrollment because we were interested to see, among those who 
enrolled in college math courses, whether the college math GPA earned by QW1 students was 
lower or higher than, or comparable to, the matched comparison students. The results 


CARNEGIE POST-QUANTWAY SUCCESS 8 


(presented in Table 2) indicate a significant interaction effect of QW1 and college math 
enrollment (as well as significant main effects of these two variables). For ease of 
interpretation, we transformed the model-based results into group mean GPAs and found that 
the GPAs for the QW1 group and the matched comparison group were comparable at 2.22 and 
2.06, respectively. Significant variation among colleges was also observed (0.010); however, the 
variation pertained to the intercept (grand mean of GPA), but not the slope (effect size of QW1 
on GPA).° Overall, the results derived from college math enrollments and GPA suggested that 
QW 1 students were more likely than comparable students who enrolled in traditional 
developmental math sequences to enroll in a college math course in the following year and to 
demonstrate comparable performance in college math. 


Subgroup Analyses 

To examine possible differential effects of QW1 by sex and race/ethnicity subgroups, we 
constructed four-level HLMs similar to those described above. In this subgroup analysis we 
applied effect coding to the grouping variables in order to directly represent both main and 
interaction effects in the outcome. The reference categories were female and White (since that 
demographic group was the largest in the sample), and each of these was coded as -1.’ We 
excluded cases where sex hadn’t been specified. Figures 3 and 4 present the model-based 
results transformed back into their natural metrics of proportion of students enrolling in college 
math courses and earning an associated GPA, respectively. Positive effects of QW1 were 
observed for all subgroups. More specifically, Black and Hispanic male students, as well as Black 
female students, exhibited the largest increase in college math enrollment rates relative to the 
corresponding subgroups of matched comparison students, suggesting that they benefited 
most from QW1. Each subgroup of students also showed a college math GPA comparable to its 
matched comparison students. 


Discussion 

This study is a sequel to the previous causal-analytic study (Yamada et al., 2016), 
assessing QW1’s effectiveness on student college math achievement by tracking the propensity 
score matched students throughout the year after QW1 enrollment. We measured two follow- 
up outcomes: college math enrollment as engagement and an associated college math GPA as 
performance. The results revealed that half of QW1 students enrolled in college math courses 
within a year, whereas only one-third of their counterparts who followed traditional 
developmental math sequences did so, and that QW1 students demonstrated a comparable 
GPA. It is plausible that the former may be a consequence of QW1’s emphasis on strengthening 
growth mindset of students as mathematical learners and doers, enhancing their sense of 
belonging in a mathematical environment, and helping them develop the confidence and 
tenacity to grapple with the complex language of mathematics. Accordingly, QW1 students 
persisted and engaged more in college-level mathematics in the subsequent year. 


° The obtained result does not necessarily mean no variation in QW1 effect size among colleges. Data analyzed in this study were from the 
relatively small number of colleges (9 colleges), and it might be possible that we would detect significant, meaningful variation with more 
colleges. 

” Because we included interaction terms of those two sets of effect-coded demographic groups, the actual reference group in the HLM models 
was white female. 


CARNEGIE POST-QUANTWAY SUCCESS 


Another focus of our analysis was on variation in QW1 effect among different colleges, 
faculty, and student subgroups. To advance efficacy reliably at scale, QW1 should work for 
diverse student populations across a wide range of classroom and institutional contexts. 
Therefore, it is critical to learn from variation and find a way to reduce variation in desired 
outcomes (Bryk, Gomez, Grunow, & LeMahieu, 2015). Our results indicated that QW1 effects 
were positive across all sex and race/ethnicity subgroups of students, as well as nearly all 
classrooms and colleges, suggesting that QW1 reduced variability in student outcomes and 
promoted positive outcomes across subgroups. Interestingly, one college identified as a 
positive deviant in the previous study demonstrated the similar pattern of success in college 
math achievement: it outperformed other colleges and maintained high performance across all 
the classrooms in the college (see College 8 in Figures 1 and 2). This college may make a great 
case from which to learn how effectively and reliably it adapted QW1 into its local context. We 
can then spread its practices to other institutions nationwide and facilitate network-wide 
improvement through improvement science embedded in the networked improvement 
community (Bryk et al., 2015). 

Finally, it is worth mentioning a future direction for further exploration. We are 
currently analyzing data obtained from the National Student Clearinghouse. Our particular 
interest is in 2-year and 4-year degree completion rates as well as transfer rates into 4-year 
colleges of QW1 students as more distal outcomes. This analysis would further illuminate the 
extent and dimensions of QW1’s effectiveness. 
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Appendix 
List of Participating Colleges 


Atlantic Cape Community College 
Borough of Manhattan Community College 
Cuyahoga Community College 

East Georgia State College 

Madison College 

Marshall University 

Onondaga Community College 
Ridgewater College 

Rockland Community College 

Sinclair Community College 

South Georgia State College 

University of North Georgia, Gainesville 
University of Washington, Bothell 
Westchester Community College 
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Table 1. 


Model-Based Estimation of QW1 Effect on College Math Enrollment Rate 


Fixed effect 
Intercept 

W12 

$12 

F12 

$13 
QW1 

W12 

$12 

F12 

$13 
Propensity score 


Random effect at level 4 (college) 


Intercept 
Qw1 


Random effect at level 3 (faculty) 


Intercept 
Qw1 


Coefficient 


-1.11 
0.63 
-0.08 
0.00 
-0.52 
0.85 
-0.24 
0.24 
0.31 
-0.02 
0.19 


Variance 


0.219 
0.299 
Variance 


0.013 
0.094 


SE 


0.17 
0.17 
0.09 
0.08 
0.08 
0.21 
0.34 
0.18 
0.16 
0.14 
0.03 


df 
8 


df 


44 


t 
-6.68 
3.62 
-0.89 
-0.01 
-6.44 
3.99 
-0.71 
1.32 
1.96 
-0.15 
6.70 


va 


154.40 
66.07 


p 
<0.001 
<0.001 
372 
.990 
<0.001 
.001 
.476 
.188 
.050 
878 
<0.001 


p 


12 


Odds ratio 
0.33 
1.87 
0.92 
1.00 
0.59 
2.33 
0.79 
1.27 
1.37 
0.98 
1.21 


Correlation 


-0.55 


Correlation 


-0.81 
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Table 2. 
Model-Based Estimation of QW1 Effect on College Math GPA 
Fixed effect Coefficient SE t p 
Intercept 0.57 0.04 14.50 <0.001 
W12 -0.10 0.08 -1.18 .239 
$12 -0.13 0.04 -2.80 .006 
F12 -0.11 0.04 -2.79 .006 
$13 -0.27 0.03 -7.89 <0.001 
QW1 0.04 0.01 3.78 <0.001 
W12 0.10 0.06 1.76 .078 
$12 0.08 0.03 2.59 .010 
F12 0.07 0.03 2.52 .012 
$13 0.05 0.03 1.91 .056 
College math enrollment 1.03 0.01 87.80 <0.001 
W12 -0.19 0.06 -2.99 .003 
$12 -0.13 0.03 -3.74 <0.001 
F12 -0.07 0.03 -2.33 .020 
$13 -0.03 0.03 -0.91 363 
QW 1 x College math enrollment 0.04 0.01 3.87 <0.001 
W12 0.10 0.06 1.63 .102 
$12 0.06 0.03 1.84 .065 
F12 0.05 0.03 1.75 .080 
$13 0.06 0.03 2.00 .046 
Propensity score 0.07 0.01 4.75 <0.001 
Random effect at level 4 (college) Variance Df a p 
Intercept 0.010 8 77.95 <0.001 
Random effect at level 3 (faculty) Variance Df x p 
Intercept 0.002 44 60.36 0.051 
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QW1 effect (in logits) 


4 10 3 6 7 5 2 8 


HB QW1 effect by college Upper bound Average effect Lower bound 


Figure 1. Variation in QW1 effect on college math enrollment rates among colleges. 
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QW1 effect (in logits) 


@ QW1 effect by faculty Upper bound = Average effect Lower bound 


Figure 2. Variation in QW1 effect on college math enrollment rates among faculty members. 
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Figure 3. Model-based college math enrollment rates by sex and race/ethnicity. 
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Figure 4. Model-based college math GPA by sex and race/ethnicity. 
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